Prevalence and appropriateness of in-person versus not-in-person ambulatory antibiotic prescribing in an integrated academic health system: A cohort study

Objectives Ambulatory antibiotic stewardship generally aims to address the appropriateness of antibiotics prescribed at in-person visits. The prevalence and appropriateness of antibiotics prescribed outside of in-person visits is poorly studied. Design and setting Retrospective cohort study of all ambulatory antibiotic prescribing in an integrated health delivery system in the United States. Participants Antibiotic prescribers and patients receiving oral antibiotic prescriptions between January 2016 and December 2019. Main outcome measures Proportion of antibiotics prescribed with in-person visits or not-in-person encounters (e.g., telephone, refills). Proportion of prescriptions in in 5 mutually exclusive appropriateness groups: 1) chronic antibiotic use; 2) antibiotic-appropriate; 3) potentially antibiotic-appropriate; 4) non-antibiotic-appropriate; and 5) not associated with a diagnosis. Results Over the 4-year study period, there were 714,057 antibiotic prescriptions ordered for 348,739 unique patients by 2,391 clinicians in 467 clinics. Patients had a mean age of 41 years old, were 61% female, and 78% White. Clinicians were 58% women; 78% physicians; and were 42% primary care, 39% medical specialists, and 12% surgical specialists. Overall, 81% of antibiotics were prescribed with in-person visits and 19% without in-person visits. The most common not-in-person encounter types were telephone (10%), orders only (5%), and refill encounters (3%). Of all antibiotic prescriptions, 16% were for chronic use, 15% were antibiotic-appropriate, 39% were potentially antibiotic-appropriate, 22% were non-antibiotic-appropriate, and 8% were not associated with a diagnosis. Antibiotics prescribed in not-in-person encounters were more likely to be chronic (20% versus 15%); less likely to be associated with appropriate or potentially appropriate diagnoses (30% versus 59%) or non-antibiotic-appropriate diagnoses (8% versus 25%); and more likely to be associated with no diagnosis (42% versus <1%). Conclusions Ambulatory stewardship interventions that focus only on in-person visits may miss a large proportion of antibiotic prescribing, inappropriate prescribing, and antibiotics prescribed in the absence of any diagnosis.

However, antibiotics can be prescribed without in-person visits, for example when patients and clinicians communicate via telephone call or electronic patient portal messages. Antibiotic prescribing at these "not-in-person encounters" are overlooked in many stewardship initiatives and in research. Not-in-person encounters may also have lower quality because of an inability to do a physical examination. Prior publications have examined the appropriateness of outpatient antibiotic prescribing across all settings using claims data, but these studies did not distinguish between antibiotics prescribed with and without in-person encounters [5,[21][22][23].
Our objectives were, among all antibiotics prescribed within a large integrated health system using an electronic health record (EHR), to 1) assess the prevalence of not-in-person antibiotic prescriptions and 2) assess the appropriateness of antibiotic prescribing across both inperson and not-in-person encounters, thus providing a comprehensive picture of outpatient antibiotic prescribing [24]. We used EHR data because they have more detailed information on encounter type compared with claims data. Moreover, the indication for antibiotics can more readily be ascertained using EHR data, using problem lists and diagnosis codes.

Study design and setting
We conducted a retrospective cohort study of all ambulatory, oral antibacterial antimicrobial prescriptions ordered in the EHR between January 1, 2016 and December 31, 2019 by clinicians at Northwestern Medicine, prior to the COVID pandemic and the introduction of synchronous, remote visits. Northwestern Medicine is a large, integrated academic health delivery system with hundreds of sites, about 4000 physicians, and 30,000 staff in the Chicago area (Illinois, USA). Northwestern uses a single EHR (Epic; Verona, WI). Northwestern aggregates clinical data in the Northwestern Medicine Enterprise Data Warehouse (NMEDW) [25]. The Northwestern University Institutional Review Board approved this study with a waiver of informed consent for retrospective review of EHR data. Patients and the public were not involved in the design, conduct, reporting, or dissemination of this work.

Antibiotics
We used an EHR-based medication grouper to identify oral antibiotic prescriptions (S1 File). We included antibiotics prescribed by clinicians of all specialties to patients at an ambulatory clinic, regardless of patient age; whether the patient lived at home, at a care facility, or elsewhere; or any other patient factor. We excluded non-oral antibiotics and methenamine. The unit of analysis is the prescription: individual prescribers could contribute multiple antibiotic prescriptions and individual patients could be associated with multiple antibiotic prescriptions.

In-person visits or not-in-person encounter types
The EHR organizes units of work into "encounters" of differing types such as clinic visits, telephone calls, electronic patient portal messages, and refills, among others. All prescriptions are generated within a defined encounter type in the EHR. Diagnoses are not required within all encounter types. For example, at least one diagnosis was required for an in-person encounter, but not for a telephone or patient portal encounter.
We classified encounters as either in-person or not-in-person. We considered the following encounter types not-in-person: telephone, patient portal, orders only, refill, and other. Orders only encounters are created when clinicians enter a patient's record solely to enter orders (e.g., prescriptions) without any other activity or documentation. Any prescriptions entered in this manner are, by definition, non-visit-based since a visit would generate a different type of encounter in the electronic record. During the study period, which predated the COVID-19 pandemic, we did not have separate synchronous telephone, synchronous video, or asynchronous e-visit (e.g., questionnaire-based) visit types.
We validated this classification with manual chart review. Two study team members (AG, TB) independently reviewed 54 charts with "other" encounter classification type to determine whether any of these rarely used documentation mechanisms could be classified as in-person visits, but all "other" types were determined to be not-in-person encounters.

Appropriateness
In the Northwestern EHR, the encounters in which antibiotics are ordered can be associated with zero, one, or more than one diagnosis codes. We previously defined all 94,249 International Classification of Diseases, 10 th Edition, Clinical Modification (ICD-10-CM) diagnosis codes as always, sometimes, or never justifying antibiotic prescribing [5]. Using a slight modification of this scheme, we classified all antibiotic prescriptions into 5 mutually exclusive groups: chronic antibiotic use, antibiotic-appropriate, potentially antibiotic-appropriate, nonantibiotic-appropriate, or not associated with any diagnosis (S1 Data). Our scheme erred on the side of being lenient to prescribers, increasing the likelihood that an antibiotic prescription would be associated with an antibiotic-appropriate diagnosis (e.g., we would consider the antibiotic appropriate if there were both antibiotic-appropriate and non-antibiotic-appropriate codes present).
Chronic antibiotic use. We identified 690 ICD-10-CM codes that could be associated with a chronic infection or might be indications for chronic antibiotic use (e.g., chronic osteomyelitis, acne, cystic fibrosis, emphysema) [22,23]. We examined diagnosis codes and problem list diagnoses present in the 6 months prior to antibiotic prescribing and, if present, considered those to be for chronic antibiotic use, regardless of the nature of any other associated diagnosis codes during this lookback period.
Antibiotic-appropriate, potentially antibiotic-appropriate, non-antibiotic-appropriate. For the remaining antibiotics, we compiled all encounter diagnoses on the same day as the antibiotic prescription, regardless of whether diagnoses were coded at in-person versus notin-person encounters, and regardless of which prescriber coded the diagnosis (i.e., the antibiotic could be prescribed and the diagnosis applied by different clinicians on the same day). Additionally, we compiled all diagnosis codes coded by the antibiotic prescriber up to 21 days later, as our manual chart review revealed instances in which a clinician would initiate an encounter, apply a diagnosis, and, within that same encounter, prescribe an antibiotic up to 21 days later. A subsequent analysis with the data from the present study revealed subtle differences in results using different durations between the antibiotic prescription and diagnosis codes [26].
If at least one of the 9,495 "always" diagnosis codes was present, we classified the prescription as antibiotic-appropriate. If at least one of the 11,143 "sometimes" diagnosis codes but no "always" code was present, we classified the prescription as potentially antibiotic-appropriate. If at least one "never" code but no "always" or "sometimes" code was present, we classified the antibiotic prescription as "non-antibiotic-appropriate." No diagnosis. If there was no diagnosis code, we considered the prescription associated with no diagnosis.
Within each category, we examined the 5 most common diagnoses reported in the prior 6 months for chronic antibiotics, and the 5 most common diagnoses reported on the same day as prescriptions classified as antibiotic-appropriate, potentially antibiotic-appropriate and non-antibiotic-appropriate.

Descriptive variables
Antibiotic prescriptions were the unit of analysis. In examining differences between in-person and not-in person prescriptions, we examined clinic, patient, and clinician characteristics. We defined a clinic as a uniquely named practice site with a unique physical address. We used registration data for patient age, gender, ethnicity, race, marital status, and insurance. We calculated patients' neighborhood household income by matching patient zip codes to 2013-2017 American Community Survey data. We used the EHR to calculate the number of patient comorbidities, prescriptions, clinic visits, emergency department visits, hospitalizations, and whether the patient had a listed primary care physician. For clinicians, we used the listing in the EHR to determine clinician type (e.g., physician, physician assistant, etc.).
We classified physician specialty as primary care, medical specialty, surgical specialty, and other. We calculated the number of years in practice by subtracting the year of medical school graduation from 2019. We calculated ambulatory clinical full-time equivalence by dividing the annual number of half day clinic sessions (before and after 12PM) during which clinicians had at least 3 patient encounters by total half days of employment. Our calculation only included time in ambulatory clinic and did not include inpatient service, operating time, administrative, teaching, research, or other activities. We used the EHR to measure the number of total prescriptions and number of antibiotic prescriptions written during the study period.

Statistical analysis
We used descriptive statistics to describe the cohort of prescriptions, patients, and prescribing clinicians, to calculate the prevalence of not-in-person antibiotic prescriptions, and to calculate the proportion of not-in-person and in-person prescriptions assigned to each of the 5 mutually exclusive appropriateness categories. Because the large sample size makes clinically insignificant differences statistically significant, the lack of independence among some comparisons, and interest in taking an all-antibiotic prescribing stewardship perspective, we did not perform statistical significance testing, clustered analyses, or adjusted analyses. Rather, we considered absolute differences of 5% or greater to be clinically significant [22,23]. All analyses were performed using SAS v9.4 (SAS Institute Inc., Cary, NC).

In-person versus not-in-person prescriptions
Although the groups were not independent, patients who received any not-in-person antibiotic prescriptions were more likely to be older, women, married/partnered, have Medicare, take more medication, make more visits, have a primary care physician, and be hospitalized during the study period compared with patients who received any in-person antibiotic prescription ( Table 1). Seventy three percent of clinicians prescribed both in-person and not-inperson antibiotics during the study period; 27% prescribed only one or the other.
Over the four-year study period, 81% (n = 578,130) antibiotics were prescribed with an inperson encounter and 19% (n = 135,927) with a not-in-person encounter ( Table 2). The most common types of not-in-person encounters in which antibiotics were prescribed were telephone (10% of all antibiotics prescribed with not-in-person encounters), orders-only (5%), and refill (3%) encounters.
Antibiotic appropriateness. The 5 most common diagnoses in each appropriateness category are listed in the footnote of Table 2. Across all encounter types, chronic antibiotic use accounted for 16% of all antibiotic prescriptions, while antibiotic-appropriate and potentially antibiotic-appropriate prescriptions collectively accounted for 54% ( Table 2). Non-antibioticappropriate diagnoses accounted for 22% of antibiotic prescriptions. Eight percent of antibiotic prescriptions were associated with no diagnosis. Thus, 30% of antibiotics were either nonantibiotic-appropriate or not associated with any diagnosis.
Antibiotics prescribed in not-in-person encounters were more likely to be chronic (20% versus 15% compared to antibiotics prescribed with in-person visits). Among not-in-person encounter types, antibiotic prescriptions were most likely to be for chronic use among refill (29%) and patient portal (22%) encounters. Not-in-person antibiotics were less likely to be associated with appropriate or potentially appropriate diagnoses (30% versus 59% for in-person visits). Not-in-person antibiotics, depending on the specific encounter type, were variably less likely to be associated with antibiotic-appropriate diagnoses. Not-in-person antibiotics were less likely associated with potentially-antibiotic-appropriate diagnoses (20% versus 43%). Except for the relatively rare "other" not-in-person encounters, not-in-person encounters were less likely to be associated with non-antibiotic appropriate diagnoses (8% versus 25%). Proportion of all prescriptions accounted for by antibiotics, Median % (IQR) 6 (3, 11) 6 (3, 11) 6 (3, 10) a. Percentages may not add up to 100% because of rounding. b. There were 47,460 patients (14% of all patients) in both the "any in-person" and "any not-in-person" groups. See Methods for the rationale for not performing statistical testing between groups.
c. 20 patients with missing data for gender. d. There were 1743 clinicians (73% of all clinicians) in both the "any in-person" and "any not-in-person" groups.
e. 5 clinicians with missing data for gender.
f. APN is Advanced Practice Nurse. NP is Nurse Practitioner. g. There were 1149 physicians (82% of all physicians) in both the "any in-person" and "any not-in-person" groups.
i. 203 physicians with missing data for years since medical school graduation https://doi.org/10.1371/journal.pone.0289303.t001 c. The 5 most common chronic diagnoses within the 6 months prior to prescribing were chronic sinusitis, unspecified; acne vulgaris; chronic obstructive pulmonary disease, unspecified; rosacea, unspecified; and chronic rhinitis.
d.The 5 most common same-day antibiotic-appropriate diagnoses were urinary tract infection, site not specified; streptococcal pharyngitis; acute cystitis without hematuria; acute cystitis with hematuria; and pneumonia, unspecified organism. e. The 5 most common same-day potentially appropriate infection-related diagnoses were acute sinusitis, unspecified; acute pharyngitis, unspecified; acute maxillary sinusitis, unspecified; acute pansinusitis, unspecified; and acute frontal sinusitis, unspecified.
f. The 5 most common same-day non-antibiotic-appropriate diagnoses were acute upper respiratory infection, unspecified; acute bronchitis, unspecified organism; bronchitis; cough; and encounter for immunization. https://doi.org/10.1371/journal.pone.0289303.t002 Not-in-person encounters were more likely associated with no diagnosis code (42% versus <1%). Said another way, 58% of not-in-person encounters had a diagnosis whereas 99% of inperson encounters had a diagnosis.

Discussion
In this study of four years of antibiotic prescribing data within an integrated academic health system, we found that 19% of ambulatory antibiotics were prescribed with not-in-person encounters. Antibiotics prescribed without in-person visits were more likely for chronic diagnoses; less likely to be associated with appropriate or potentially appropriate diagnoses; and more likely to be associated with no diagnosis. These findings suggest that ambulatory stewardship interventions that focus only on in-person visits may miss a large proportion of antibiotic prescribing, inappropriate prescribing, and antibiotics prescribed in the absence of any diagnosis. The absence of a diagnosis may say more about inappropriate documentation than inappropriate antibiotic prescribing. However, if one considered the absence of a diagnosis to be inappropriate prescribing, 25% of in-person antibiotics were inappropriate and 50% of notin-person antibiotics were inappropriate. Our findings are particularly important considering the COVID-19 pandemic. During the pandemic, there were shifts in visit numbers: in-person visits have become less frequent and use of not-in-person encounters have become more common [27]. At the study health system, for example, there are now many more synchronous telephone and video visits that require a billing code. There were also pandemic shifts in antibiotic prescribing [28][29][30].
Familiarity with remote care and improved reimbursement for synchronous, remote visits has the potential to improve all-ambulatory-antibiotic stewardship. Alternatively, increased remote care has the potential to be associated with decreased antibiotic prescribing quality. Analyses of all-antibiotic prescribing, appropriateness, and visit type should be repeated and the results validated in the COVID pandemic and post-pandemic periods. Given the shifts in visit types and antibiotic prescribing, a continued focus only on antibiotic prescribing during in-person visits or solely on telehealth visits is less and less justifiable. Similarly, studies that only focus on telemedicine may only capture a fraction of overall antibiotic prescribing at a health system [31][32][33].
Our appropriateness findings are similar to what Fleming-Dutra and colleagues reported using nationally representative survey data from the United States. After sampling from ambulatory visits with oral antibiotic prescriptions they found an estimated annual antibiotic prescription rate of 506 per 1000 population and an estimated 353 (70%) were likely clinically appropriate [34]. We have conducted studies using claims data and found between 26% and 29% were not associated with a recent diagnosis code [5,21], quite a bit higher than the 8% from the present study. Regarding not-in-person encounters, we previously found that about 30% of antibiotic fills were "non-visit-based" [22,23] which is more than the 19% of not-inperson encounters in the present study. A key difference is that in the prior studies we had a more restrictive definition of chronic antibiotic prescribing, based on long-term dispensing. In the present study, to be more lenient to clinicians, we initially associated antibiotic prescriptions with chronic conditions based on 6 months of diagnoses. Claims and EHR data might differentially detect antibiotics: antibiotics might be prescribed in the EHR, but not generate a claim (i.e., the patient paid cash or did not pick up the antibiotic) or clinicians could circumvent EHR prescribing by calling a pharmacy directly. Compared to claims data, EHR data may be more likely to demonstrate the relationships among prescribers, antibiotics, sites of care, and diagnoses. Claims data may rely on different information sources for antibiotic claims, prescriber information, and diagnosis codes and overstate the "inappropriateness" of antibiotic prescribing relative to EHR data.
Our study is subject to several limitations beyond it being conducted prior to the pandemic. First, it was conducted within a single integrated health delivery system which may limit generalizability. Second, the data we analyzed were dependent on clinician diagnosis code selection within the EHR which is sometimes an incomplete reflection of actual care. Third, we were limited to prescribing data (e.g., clinician behavior) and did not have data regarding if or when patients filled and took their prescribed antibiotic (e.g., patient behavior).
Fourth, in determining appropriateness, we erred on the side of leniency. For example, we considered all antibiotic prescribing for chronic conditions to be acceptable (e.g., emphysema, chronic sinusitis, and chronic rhinitis) even though the prescribing may not have been clinically appropriate. We also forgivingly assumed that the most antibiotic-appropriate diagnosis was the reason for the antibiotic prescription, did not require the antibiotic class to be appropriate for a given condition, and could use a single diagnosis code to justify the appropriateness of multiple antibiotics. Thus, our study's estimate of inappropriate prescribing is undoubtedly a lower bound.
Fifth, our approach assumed that antibiotic prescriptions on different days resulted from a separate episode of care, but this may not always be the case. For example, if a follow-up telephone visit by a patient or pharmacist to clarify an antibiotic written at an in-person visit resulted in a new antibiotic prescription, this would have been counted as two separate prescriptions. However, any resulting bias is likely minimal, as only 2% of antibiotic prescriptions had another antibiotic prescription in the subsequent 1 to 3 days. Sixth, it was beyond the scope of the current study to examine prescriber-level variability in antibiotic appropriateness, including whether certain prescribers disproportionately accounted for not-in-person antibiotic prescribing that lacked diagnosis codes or whether prescribers had different patterns of appropriateness across their in-person and not-in-person encounters. Seventh, direct comparisons of appropriateness between prescriptions written with in-person visits versus not-in-person encounters were limited by the frequent lack of diagnosis codes among prescriptions at not-in-person encounters. However, the lack of an associated diagnosis code could itself be viewed as inappropriate or a quality problem that would limit the effectiveness of ambulatory stewardship programs. Additional research is needed to examine the rationale of antibiotics prescribing not associated with a diagnosis code.
In summary, to decrease unnecessary antibiotic prescribing, improve patient safety, and slow the development of antibiotic-resistant bacteria, ambulatory antibiotic stewardship interventions should comprehensively measure all antibiotic prescribing, not just prescribing at inperson visits. The importance of considering prescribing across all encounters may only grow if ongoing shifts towards remote care continue.